ON QUADRATIC DISTINCTION OF AUTOMORPHIC SHEAVES 



YIFENG LIU 

Abstract. We prove a geometric version of a classical result on the characterization 
of an irreducible cuspidal automorphic representation of GL n (A.E) being the base 
change of a stable cuspidal packet of the quasi-split unitary group associated to the 
quadratic extension E/F, via the nonvanishing of certain period integrals, called 
being distinguished. We show that certain cohomology of an automorphic sheaf of 
GL„,x' is nonvanishing if and only if the corresponding local system E on X' is 
conjugate self-dual with respect to an etale double cover X' / X of curves, which 
directly relates to the base change from the associated unitary group. In particular, 
the geometric setting makes sense for any base field. 
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Introduction 

0.1. Classical point of view: distinguished representations. Let E/F be a quadratic 
extension of global fields with the Galois group {1,<t}, II an irreducible cuspidal auto- 
morphic representation of GL n (A^). We consider the following two period integrals. 



P+(# = / <P(h)dh; 
J GL„(F)\GL n ( 



P~0) = / (f>(h)u) E/F (deth)dh 

where <p is a cusp form in the space of II, GL n (A^)° is the subgroup of GL n (Ap) 
consisting of elements whose determinant has norm 1, and oje/f is t ne quadratic char- 
acter of Ap associated with E/F via global class field theory. If P + (resp. P~) is not 
identically zero on (f> 6 II, then we say II is distinguished (resp. (^/^-distinguished) 
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by GL n! p, or simply distinguished (resp. u^/p-distinguished). Assuming the central 
character of IT is distinguished, it is proved in [4] Proposition 1, that IT V = U a if and 
only if IT is distinguished (resp. distinguished or w^/p-distinguished) when n is odd 
(resp. even). Here, LT V and H a denote the contragredient representation and the a- 
twisted representation, respectively. The proof is based on previous results of Flicker 
[5], [6]. From the point of view of L- functions, II is distinguished if and only if the 
Asai L-function L(s,H, As) has a (simple) pole at s = 1 and II is w^/^-distinguished if 
and only if L(s,H <8> ^2, As) has a (simple) pole at s = 1, where f2 is any automorphic 
character of whose restriction to A F equals oje/f c f- [5]- 

The theory has a perfect geometric counterpart, in the framework of Geometric 
Langlands Program. Instead of automorphic representations, we work on automorphic 
sheaves and hence only consider the everywhere unramified case. Then we will replace 
the period integrals P + and P _ by certain complexes of £-adic sheaves on the base 
field, which now makes sense for any field k. The isomorphism IT V = U a has a per- 
fect Galois counterpart, namely the isomorphism E v = E a for the corresponding local 
system E of II. The use of poles of L-functions will implicitly appear in our proof as 
cohomology. Now we are going to state our situation and Main Theorem in more details. 

0.2. Geometrization and Main Theorem. Let k be a field (which can and will be assumed 
algebraically closed) and I a prime number invertible in k. Let X be a connected smooth 
proper curve over k and fi : X' — > X be a proper etale morphism of degree 2 with X' 
connected. Let a be the unique nontrivial isomorphism such that the following diagram 
commutes 

X' -A" 




X 



For a positive integer n, we denote Bun n (resp. Bun' n ) the moduli stack of rank-n 
vector bundles on X (resp. X'). Then the pullback of vector bundles under // induces a 
morphism jjL n : Bun n — » Bun' n . The stack Bun n (resp. Bun' n ) is a disjoint union of its 
connected components Bun^ (resp. Bun'^) parameterizing those bundles of normalized 
degree (cf. 0.8) d for d € Z, then we have /i„ : Bun^ — > Bun'^ d . Moreover, we have an 
isomorphism a n : Bun' n — > Bun' n , induced by the pullback under a. 

For a local system E on X' of rank n, we let E y be its dual system and E a = o~*E. 
If E is irreducible, then the geometric Langlands correspondence (proved by Drinfeld 
[3] for n = 2, formulated by Laumon [12], [13] and proved by Frenkel-Gaitsgory-Vilonen 
for general n [9]) associates E aQ< perverse sheaf on Bun' n , denoted by Autg, which is 
irreducible and nontrivial on each Bun'^ and satisfies the Hecke property with respect 
to E. If we denote the trivial rank-one local system, then we have a canonical 
decomposition /x*Q^ = Qe ® for a degree 2 rank-one local system L M on X. The 
determinant of vector bundles induces a morphism det : Bun n — > Buni = Picx pre- 
serving degree. For a rank-one local system L on X, we define Tl = det*A^, where 
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Al is the local system placed at degree zero shifted from Aut^. For simplicity, we let 
T M := T Lm . 

We will define in Section 1 one of the primary objects in this paper, the Asai local 
system As(E), which is a local system of rank n 2 on X. It is constructed by the 
geometric analogue of the classical method, called multiplicative induction or twisted 
tensor product cf. [18] Section 7, for the Asai representation of the Galois group. The 
following is our Main Theorem. 

Main Theorem. Let notations be as above and E an irreducible local system on X' of 
rank n, consider the following statements: 

(a) . E v ~ E a ; 

(b) . DAut_e — <7*Aut_E, where D denotes the Verdier duality; 

(c) . Rr^Bun'O, Aut£ ® a*Aut E ) + 1 ; 
(d+). Q e C As(-E); 

(e+). Rr c (Bun°,^Autij) / 0; 

{d-). L„ C As(E); 

(e"). Rr c (Bu<, M^AutE ® T„) ^ 0. 

Then, we have the following equivalence 

(a) (6) ( C ) or (<T) 

and 

(d+) <=► (e+); (cT) <=> (e"). 



The theorem has the following corollaries on the direct image complexes which are 
not easy to see directly. 

Corollary 0.3. (1) We have 

Rr c (Bun°,/i;Aut £ ) = <^ RT(Bun£, ji* Aut^) = 0, 

Rr c (Bun° , /i* Aut E ® Tf,,) = ^> Rr(Bun° , fi* n Aut E ® T M ) = 0; 

Between Rr c (Bun° , /i* Auts) and Rr c (Bun° , ^* AutE ® T M ), i/iere is mosi one 
which is nontrivial and there is one, if and only if E v ~ E a . 

0.4. Mirabolic calculation. The proof of Main Theorem relies on the computation of 
the direct image complex of certain sheaves on the moduli stack corresponding to the 
mirabolic subgroup of GL n . Let M n (resp. M' n ) be the moduli stack classifying pairs 
si) (resp. (^#', s'j)), where <M (resp. is an object in Bun n (resp. Bun' n ) 

and s\ : fi^ -1 ^# (resp. : ^^T 1 is an inclusion of &x -modules (resp. 

^x'-modules). Here, fi. and stand for the sheaf of differentials and the structure 

l(Dne may complain that Rr c is not defined by Laszlo-Olsson [11] since Bun'° is not of finite type 
in general, but we only talk about the triviality of this complex which is well-defined. In fact, even the 
complex makes sense since actually the support of Aut_e on Bun'° is of finite type. 
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sheaf, respectively and means f2f m-1 . Again, the pullback of pairs (^#, s±) under 
\jl induces a morphism /i n : M n — > M' n and we have obvious morphism tt : M n — > Bun n 
(resp. tt' : M' n — > Bun' n ) by forgetting s± (resp. s^). Let (resp. M'^) be the 
inverse image of Bun^ (resp. Bun'^) under tt (resp. tt 1 ). Define W# = 7r'*Aut£; (up to 
a cohomological shift). We have the following theorem. 

Theorem 0.5. Let notations be as above and E an irreducible local system on X' of 
rank n. Let L be a rank-one local system on X, then we have for d > a canonical 
isomorphism 

RT c (M d n ,jl* n \N E ® tt*J l ) ^ RT(X^,(As(E) ® L)^)[2d] 

where X^ and (As(E) ® L)^ denote the d-th symmetric product of the curve X and 
its local system As(E) (g) L, respectively. 

The proof of the above theorem follows the same line in [15] for the geometrized 
Rankin-Selberg method due to Lysenko, which can be viewed as the geometrization of 
the well-known classical way treating the Rankin-Selberg integral of Jacquet, Piatetskii- 
Shapiro and Shalika. We modify the argument in [15] to our situation, just as the mod- 
ification of the classical argument in [4], [5]. It is interesting and useful to make these 
methodological comparison, which provides certain hint for the proof in the geometric 
counterpart. 

0.6. Connection with functoriality. Let us go back to the classical situation. As pointed 
out in [4], [6], the isomorphism II V = Yi a has important meaning on the functorial 
lifting. Let \J Uj e/f denote the quasi-split unitary group of n variables with respect 
to the quadratic extension E/F. When n is odd, the representation LT with central 
character being trivial on A^, which satisfies LT V = IT 7 should be a (stable) base change 
of a stable cuspidal packet of \J ni E/F- When n is even, the representation H satisfying 
II V = Yl a should also be a base change of a stable cuspidal packet of \J n: E/F, either an 
unstable base change or a stable base change, according to whether LT is distinguished 
or ^/^-distinguished. 

In the geometric situation. Let U n x' IX be the quasi-split unitary group with re- 
spect to X' /X, which is a reductive group over X and whose Langlands dual group 
L U n ,X'/x — GL n (Q^) x tti(X) (the action defining the semi-product factors through 
tti(X')). Since we have local systems parameterizing automorphic sheaves, it is easy to 
see the following fact. When n is odd, an irreducible local system E on X' of rank n 
is a (stable) base change (in fact, restriction) of a L U nj x' /x-l° ca l system on X if and 
only if E v ~ E a and As(A n £') = Qi cf. 1. When n is even, an irreducible local system 
E on X' of rank n is an unstable or stable base change of a L U n> x' /x-l°cal system on 
X if and only if E y ~ E a . It is a dichotomy of being unstable or stable which are 
corresponding to (d + ) or (d~) in Main Theorem. Hence, our Main Theorem provides 
a geometric/cohomological characterization of automorphic sheaves of GL nj x' which 
should correspond to the (conjectural) automorphic sheaves of U n x' IX y i a geometric 
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Langlands functoriality. 

0.7. Structure of the paper. In Section 1, we will define the so-called Asai local system, 
which is the geometric analogue of the classical Asai representation of the Galois group. 
These local systems are the main objects studied in this paper. In Section 2, we prove 
Main Theorem and its Corollary assuming Theorem 0.5. The proof implicitly uses the 
idea of poles which appear as cohomology on higher symmetric products of the base 
curve. The rest sections are devoted to prove Theorem 0.5. 

In Section 3, after recalling the definition of Laumon's sheaf and the Whittaker 
sheaf, we reduce Theorem 0.5 to certain formula (see Proposition 3.2) relating direct 
images of Whittaker sheaves and symmetric products of constructive sheaves on the 
base curve. Section 4 and Section 5 are responsible for the proof of this formula. 

0.8. Notations and Conventions. 

We fix a field k (which can and will be assumed algebraically closed) and a prime 
I invertible in k. Fix an algebraic closure of Q^. We work with Artin stacks, i.e., 
algebraic stacks in the smooth topology cf. [14], locally of finite type over k. In the 
main body of the paper, we will assume k has positive characteristic p and work in the 
derived category of unbounded complexes of (Q^-sheaves with constructible cohomology 
on an Artin stack X locally of finite type over k, in the sense of [11], which we denote 
by D(X). The operations /*, /•, /*, /i, <g) are only applied to locally bounded complexes 
and understood in the derived sense. For K £ D(X), we let H* K be its i-th cohomology 
sheaf with respect to the usual t-structure. In particular, we denote the constant 
sheaf placed in degree 0. We fix a nontrivial additive character tp : ¥ p — >■ Q^ x and 
denote AS,/, the corresponding Artin-Schreier sheaf on the fc-affine line G a All the 
results and proof work perfectly in the case when k has characteristic 0, after replac- 
ing D(X) by the derived category of ^-modules with holonomic cohomology and the 
Artin-Schreier sheaf AS^, by the ^-module "e x ". 

We say a stack X classifies something, we always mean that it is an Artin stack locally 
of finite type over k such that for any fc-scheme S and morphism S — > S' , the groupoid 
Hom(S', X) and the functor Hom(S",X) — > Hom( l S', X) are clearly understood. For ex- 
ample, for the stack M n defined above, objects in Hom(5, M n ) are the pairs (^#s, si,s) 
where is a vector bundle on S x X of rank n and s\ : s '■ ^s^f^ - IS an inclu- 

sion of (^sxx-modules such that the quotient J%s/ 1™ s i,s is S-flat; morphisms between 
(..-^5, si t s) and {<As,ti,s) are isomorphisms / : j^s — > such that / o s± t s = ti,s- 

We identify the lattice of coweights of GL m : A m with Z m . Let 

A m , P os = {A = (Ai, A m ) e A m | Ai H h A; > 0,i = 1, ...,m}; 

A m ,cff = {A = (Ai,..., A m ) e A m | Ai > 0,i = l,...,m} C A miPOS ; 
A m ,+ = {A = (Ai, A m ) G A m | Ai > A2 > • • • > A m > 0} C A m e fj; 
A m ,_ = {A = (Ai, A m ) e A m I < Ai < A 2 < • • • < A m } c A m>cff . 



6 



YIFENG LIU 



For d G Z, we let = {A = (A 1 ,...,A m ) G A m ,? | Ai + • • • + A m = d} for ? = 

0, pos, eft, +, — . Finally, we denote 6 m the group of m-permutations. 

For a connected smooth scheme S over k and d > 0, we define its d-th symmetric 
product as the quotient of S d under the action by ©^ (S^ = Specfc). For a local 
system E on 5 of rank n, we let E^ = (sym ! .E K ' rf ) e ' i be the d-th symmetric product of 
E, where sym : S d — > is the natural projection. Hence E^[d ■ dimS 1 ] is a perverse 
sheaf on . It is irreducible if and only if E is. We denote by detE = A m E the 
determinant of E, where m is rank of E. 

We fix X to be a connected smooth proper curve over k, then X^ is the variety of 
effective divisors of degree d on X, i.e., collections of d closed points. For A G ^m,pos^ 
we let X x = FJ™ 1 x( Al+ - +Al ). Let X x (resp. X x ) be the closed subscheme of X x 
such that {D 1 ,D 1 + D 2 , ■■■,D 1 + • • • + D m ) G X x belongs to X\ (resp. X x ) if and 
only if < D\ < ■ ■ ■ < D m (resp. D\ > ••• > D m > 0), which is nonempty if 
and only if A G + (resp. A G A^_). They are both contained in the subscheme 

X x s := UiLi x(Xl) xX ( when Ai > 0). We denote by sym A : X x -> the 
projection to the last factor, and also for its restriction to X x s , X x or X x . 

Let X be as above and ^# a coherent sheaf on X, its degree deg^# is normalized 
to be its usual degree minus m(m — l){g — 1), where m = rk^# is the rank of ^ and 
g is the genus of X. Hence deg®™^ Q. l x = 0. 



1. ASAI LOCAL SYSTEMS AND CONJUGATE SELF-DUALITY 

1.1. Asai local systems. Consider an etale morphism [i : X' — > X of degree I > 1 with 
X' being connected, and a local system E of rank m > 1 on X' . Since /i is etale, we 
have a canonical isomorphism 



X ( x' xx-^xxX' -a]/©!. 



I times 

Composing with the closed embedding X' xx ■ ■ ■ xx X' X'\ we get a morphism 

^(D . X — > X' {1) - A 
where we abuse the notation A for various kinds of diagonals. 
Definition. We define the Asai local system As(E) by 

As(E) := M «* (sym,^') 6 ' = (rf^ 

Since the symmetrization morphism sym : X' 1 -> X' {1) is etale with Galois group &i 
away from A, As(E) is a local system on X of rank m l . 



Remark. Surprisingly, the construction of Asai local systems is canonical, not like the 
construction of the classical Asai representation. For the later one, we need to choose a 
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set of representatives for the left coset of tti(X',x') in iri(X,x) and show that the iso- 
morphism class of representation obtained from the process of multiplicative induction 
is independent of the choice cf. [18] Section 7. 

From now on, we will only consider the case 1 = 2. Let a : X' — > X' be the unique 
nontrivial isomorphism such that /j, = fj, o a, and E a = a*E. Since I = 2, we have 
H*Qe — Qe © Ln for some rank-one local system L M on X. 

Lemma 1.2. (%) VFe feowe As(.E v ) ~ As(£) v and As(^) ~ As(E a ). 

(2) There is a canonical isomorphism fJ,*(E ® S CT ) — > As(E) © As(E) <8> L^. 

Proof. (1) is straightforward. For (2), consider the following diagram 

X' — — >■ X' x X' - A 

At sym 

X - -X'( 2) -A 

which is Cartesian. Over X' x X' — A, we have a canonical isomorphism sym* 
E® 2 . Hence 

^*As(£) ~ M * M (1) *^ (2) - (!> <r)*sym*£( 2 ) (1, 2 ~ E ® E a 

By the projection formula, we have 

£ CT ) ~ As(£) ® ^*Q^ ~ As(£7) © As(E) ® L M . 

□ 

1.3. Conjugate self- duality. We make the following definition as the geometric analogue 
of the classical representation theory cf. [10]. 

Definition. Let the situation be as above, we say E is conjugate self-dual if there is a 
nontrivial map b : E ® E 10 " — > between local systems, i.e., -E v ~ -E 10 ". 

Let 6 CT : E ® E a ~ E a ® E = a*(E ® E 10 ") ^4 ct*Q7 = be the composition of 
o~*b with the transposition of two factors. Then we say E is conjugate orthogonal (resp. 
conjugate symplectic) if moreover, we have b a = b (resp. b a = —b) and denote c(E) = 1 
(resp. c(E) = —1). 



It is easy to see that if E is irreducible and conjugate self-dual, then it is either 
conjugate orthogonal or conjugate symplectic, and c(E) = c(E y ) = c(E a ). Its deter- 
minant detE is also conjugate self-dual and c(detE') = c{E) m where m is the rank of 
E. We have the following criterion. 

Lemma 1.4. For irreducible conjugate self-dual local system E, c{E) = 1 (resp. c{E) = 
—1) if and only if As(E) contains <Qi (resp. L^). 
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Proof. It is obvious that we have dichotomy on both sides, hence we only need to show, 
for example, that Qe C As(E) implies c(E) = 1. By duality and adjunction, we have 

(sym,^ 2 ) 62 ^ 2 )^/^ 

on X'^ — A. By the pullback under sym, we get a nontrivial map b' : 2 ~ 
sym*£' v ( 2 ^ — > sym*// * which makes the following diagram commutative 

E v M E y >■ E v M E v 

b' 

where the upper arrow is the transposition. By the pullback under (1,<t), we get 
b v : E v <g> E Va -> Q~e such that c(E v ) = 1, where b v = (l,cr)*&'. But this implies that 
c(E) = 1. 

□ 

1.5. symmetrization. For d > 0, we identify &d with the set of d-by-<i permutation 
matrix. Let %2d be the subset of &2d consisting of those symmetric permutation matrix 
whose diagonal entries are 0. For t = (tij) G '%2di we define 1 = ii(t) < ■ ■ ■ < id{t) and 
*a(*) < ja{t) (a = 1, ...,d) by the condition that ti a (t)j a (t) = 1- Then they are uniquely 

determined and {i a (t),j a (t) I « = l,-,<0 = {l,...,2d}. Let ^ : X® -> (X'( 2 )) (d) -> 
Xi(id) De composition map. For any i = (tij) E ^2di we define a morphism: 

nt: X /d ^X /2d x x , (2d) 

such that the i a (£)-th component of nt(xi, xa) is x a and the j a (t)-th component of 
Xit(xi, Xd) is 0(x a ). It is obvious that ttt(xi, ...,Xd) locates in X' 2d x x ,(2d) X^ d \ Define 

n := [J XH : X d := \J X' d t — ► X' M x x , (2d) X« 

^2d ^2d 

as the disjoint union over all t € 1-2d with X'^ = Moreover, let pr denotes the 

projection X' 2d x x , (2d) X^ -)■ X^ . Then we have 

Lemma 1.6. (%) TTie scheme X' 2d X x ,(2d) X^ is of pure dimension d and its irreducible 
components one-to-one correspond to schemes X' t for t 6 T2d, - 

(£,) XTie morphism n is finite and isomorphic over an open dense subscheme of X' 2d x x ,(2 d ) 
X^ d \ Since X^ is smooth, we have niQ^[<i] ~ IC where IC denotes the intersection co- 
homology sheaf on X' X x ,(2d) 

(3) We have a canonical isomorphism pr, [(E m 2d MQ}) ®n\Q}) &2d As(E)^ on 
X^ d \ where &2d acts on X' 2d naturally by permuting 2d factors. 

Proof. For any t € T2d> we define to be the subscheme of X' 2d x x ,( 2 d) X^ consisting 
of points (x\, X2d) such that Xi = cr(xj) if and only if tij = 1. Then X^ forms a 
stratification of X' 2d Xw 2t j) X^ d \ Moreover, define n* : X t — > X' t by (x±, X2d) 
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(x^m, ...,x id r t \). Then nt on* = id. Hence nt is an isomorphism onto Xt away from the 
diagonal and X' 2d x w 2 <j) X^ is of pure dimension d. Since nt is a closed immersion 
3 smoc 

once knowing (1) and (2). 



for each t and X' d is smooth, (1) and (2) follow. The statement (3) is straightforward 



□ 



2. Steps for the proof 



2.1. Proof of (d^) (e^). We first prove the equivalence between (d ? ) and (e ? ) 

for ? = +, — assuming Theorem 0.5. Since two cases are similar, we only prove the 
first case. By the Hecke property of Autg; cf. [9] Proposition 1.5, the nontriviality of 
Rr c (Bun^, ^* Aut^) is equivalent to the nontriviality of Rr c (Bun^, /i* Aut_g) for any 
d € Z. 

Recall that a vector bundle ^ on X is called very unstable if it can be written 
as ^ = © with ^ and Ext 1 ^, ^f 2 ) = 0. By [9] Lemma 3.3, for 
any line bundle Jzf on X, there exists an integer c n («Sf) such that if d > c n (Jz?) and 
Jt € Bm4(£:) with Hom^,^) ^ 0, then ^ is very unstable. Define U (resp. It') 
to be the open substack of Bun n (resp. Bun' n ) by the condition Hom(^, ^x) = 
(resp. Hom(^',0^ ; ) = 0). Then it is well-known that It n Bun^ and It' n Bun'^ 
are of finite type for any rf £ Z. Now we fix an even integer 2d > c n (f2^,) and let 
= 7r'*Aut£;[di]| Bun /2d by a cohomological shift di such that ~ vr^,W^ n (see 
2.6). Since E is irreducible, Autg is cuspidal and hence Aut£;| Bun '2d is supported on 
It' n Bun'^ by [9] Lemma 9.4. Since ^(W) C It, we have 

7ri/i*W E ~ 70/2*71-'* Aut E [di] ~ 7ri7r*/x* Aut E [di] ~ m (Q^-i^nBuo) ® Mn Aut sK]- 

By Serre duality, 7r _1 (UnBun^) is a vector bundle of some rank d 2 > over UnBun^. 
Hence restricted on It PI Bun^, we have a distinguished triangle 

vriO; WA-2d 2 ] 07 



-i 



Tensoring with /i*Aut^[di] and applying Rr c , we have a distinguished triangle 

RT c (M d n ,ll* n W E ) RT c (Bu<XAut B )[di - 2d 2 ] RT c (Bun^ /<Aut B )[di] 

Hence RT C (M^, /i* W E ) = is equivalent to Rr c (Bun^, \x* n Aut s ) = since Rr c (Bun^, M* Aut E ) 
is bounded from above. 

Now if Q7 C As(.E), then there exists d > such that RTpf^, As(£) (d) ) ^ and 
2d > c n {Q^,). Hence by Theorem 0.5 and the above argument, Rr c (Bun^, /i* Aut^) ^ 
which implies Rr c (Bun° , fi* Autg;) ^ 0. Conversely, if Rr c (Bun° , /z* Autg) 7^ 0, then 
pick up some d such that 2d > max (c n {n™,),n 2 (4g - 4)) . We have RTpfW), As(J5)W) / 
for such d which forces C As(£) or C As(£) v ~ As(£ v ). By Lemma 1.4, £ v 
is conjugate orthogonal in the later case. Hence E is also conjugate orthogonal which, 
again by the same lemma, implies that c As(E). 
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2.2. Proof of (a) (c): Rankin- Selberg. The proof of the equivalence of (a) and (c) 
follows the same line as above but replacing Theorem 0.5 by the following main result 
of Lysenko, which can be viewed as the split case (i.e., X' is disconnected). 

Theorem 2.3 ([15]). For any local systems E\, E 2 of rank n on X and any d > 0, there 
is a canonical isomorphism: 

RT c (MtW El ®W S2 ) ^Rr(X( d \(E 1 ®E 2 )W)[2d}. 



By the same argument in 2.1, we have the following. 

Lemma 2.4. For irreducible E\ and E2, Rr c (Bun° , Aut^ <S> AutE 2 ) ^ if and only if 
E 2 ~E\. 

Applying the above lemma to X = X' , E\ = E and E 2 = E a , we get the desired 
equivalence since it is easy to see that cr*Aut£ ~ Aut^ by the construction. 

2.5. Proof of the rest. The equivalence of (a) and (b) is due to the Hecke property and 
the fact that DAut^ ~ Aut^v. Lemma 1.4 implies that (a) -4=^ (d + ) or (d~). Hence 
Main Theorem has been proved. 

For Corollary 0.3, (1) is due to the fact that As(E) v ~ As(E v ) and c(E) = c(E v ); 
(2) is due to Lemma 1.4. 

2.6. Laumon's sheaf and Whittaker sheaf . Let us briefly recall the definition of Laumon's 
sheaf Lau'g. Let £ be a local system on X' of rank n and d > 0. Denote Coh^ the 
stack classifying coherent sheaves & on X of generic rank n and degree d. Inside CoIiq, 
there is an open substack CoIiq <m by the additional condition that the restriction of & 
at any geometric point has dimension < m. Denote Fl the stack classifying complete 
flags (0 = J^o C J^i C • • • C J^d) such that J^/J^-i is in CoIiq. We have morphisms 
p : F1q -} Cohg remembering ^ d and q : F1q — > (Cohj^ remembering (^/^_i)f =1 . 
Define Flo j < m = Fig x Coh d Coho )<m . Similarly, we define for X' the stacks Coh'^, 

Coh^< m , Fl'g, Fl'^< m and p' , q'. Denote div^ : CohJ ? ->• or Coh'J, X' {d) 
the norm morphism and div? = div? for ? = 0, < m. Moreover, the pullback under \x 
induces a morphism /io : Cohg -4 Coh'^. 

We have the following commutative diagram 

Coh^ ^— Fl'g Coh'J x • • • x Coh'J 



div 



div 



Xd 



and define Springer's sheaf Spr 1 ^ := p' \q'* (div xd )* E^ d with a natural action by and 
Laumon's sheaf Lau^ := Hom6 d (triv, Spr^) on CoIi'q. 
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For d > and n > 1, we introduce the stack classifying the data (^#, (sj)), where 
^# is a vector bundle on X of rank n and degree d and are injective homomorphisms 
of coherent sheaves 

Si : n £ri)+"+M — ► A^#, i = 1, n 

such that they satisfy the Plucker relations cf. [2] Section 1, [8] Section 2, [9] Section 
4, or [15] Section 4 for details. Similarly, we define the stack Q ^ classifying the data 
, (s^)) but now on X'. For our purpose, we need to introduce a twisted version of 
°4 as follows. 

Since /j, is etale of degree 2, we have a canonical decomposition /i*<^x' = @x © 
for a line bundle Jz^ on X. We also have canonical isomorphisms ^*£lx — ^X' and 
//*Jz? M ~ Let be the stack classifying the data (^#, (sj)) similar to the previous 
one but now s» are injective homomorphisms of coherent sheaves 

Si : n £-l)+~+(«-0 g) ^0+...+(i-l) _^ A i^ j i = 1} _ ^ n 

still satisfying the Plucker relations. In fact, the stack is nothing but the stack 

Bun^y defined in [8] with the T ~ GL™ -bundle 3"t on X corresponding to the m-tuple 
of line bundles 

{n n x -\n n x ~ 2 ® js? M ,...,n x ® Jf;- 2 ,^;- 1 ) . 

The pullback under /i induces a closed embedding /J n : -4 Q 2 l d . Moreover, we have 
the natural morphism 7r n : ^ Q d n -> (resp. < : Q ^ M'*) by forgetting 
(resp. sjj). Let p n = ir o 7r n : — > Bun„ (resp. = 7r' o 71^ : Q ^ — > Bun'^). 

We have morphisms c n : — > Cohg <„ (resp. : Q £ — > CoIi'q <n ) sending (^#, (sj)) 
(resp. [M' (s-))) to AM/Ims,, (resp. A n J''/ImsJ ! ) and 5„ = div< n o c n : ^Q^ -> 

(resp. < = diy| n o £ : -> Finally, let Q n = \J d >o ^t^n = U d >o 

and ^Q n = Ud>o M Q n- 

Inside M Q°, there is an open substack j : classifying the data (^#j,rj) 

where = C C • • • C is a complete flag of sub-vector bundles and 
Ti : Q x ~ l © — ^ •^il^i-x are isomorphisms for i = 1, ...,n. Similarly, one has 
and j' : Q'° <^-» q'°. We define the evaluation map ev : Q'° — > G aj fc to be sum of 
the classes in Ext^fi^T*, f^7 4_1 ) — G a ,k of the extension 

o — >- jciijei_ x — ^UJ^Li — *- .^Vi/^' — *- o 

nn-i Qn— /— 1 

for i = 1, n — 1. Define ASjJ, := j'iev*AS^ as a sheaf on Q °. 

Let Mod^ be the stack classifying modifications (./# <yK) of rank-n vector bundles 
on X such that deg(c/K/^#) = <i. We have morphisms f)<_ : Mod^ -4 Bun n (resp. 
f)_>. : Mod^ —?■ Bun n ) sending ^ Jf) to ^# (resp. <yK) and c : Mod^ ->■ Cohp 
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sending <yK) to Jf jj$ . Similarly, we define Mod'^, f)^ and c'. Consider 

the following commutative diagram cf. [9] Section 4.3 



Q 



'o 



■ 2 



Bun' 



ro 



Mod 



— 10 n 



Bun'" 



— /^ — 'n m — ' 

where 2. n = Q ,„ x Bim /o Mod„. The Whittaker sheaf \N e, u on Q n is defined by the 

formula 



(2.7) W£ jn := W B , n |^ := f,_, (h^* (AS$) ® (c' o q^)* ( Lau| ) ) [2q n + dn] 

where q n := dim At Q°. 

Notice that we have the following commutative diagram 

_, . < 



Q 



M 



pici, 



where a is the Abel-Jacobi map and 5 : -)■ Pic& (resp. 0' : M'£ -> Pic&) sends 

n(n — 1) 71(71— 1) 

(resp. jtf' ®Slyr, 2 ). Hence 



si) (resp. (^#', s^)) to the line bundle <J?<gi£l x 
there is a morphism 



^ := d' n x 5 : Q'^ x M ,2 d M d n — ► x pk . Pic^ 



and a closed embedding t := (fi^,a) : — )• X 1 ^ X pic 2d Pic^. The rest sections 
are devoted to prove the following. 

Proposition 2.8. For any Zoccd systems E on X' of rank n and L on X of rank 1, we 
have a canonical isomorphism 

S' nl (\A/| d n Kl D*A L ) t * (As(£) ® L)^ [2d] 
on X' {2d) X P . 2d Pici /or a// d > 0. 



2.9. Proof of Theorem 0.5. Assuming Proposition 2.8, the proof is immediate once 
realizing the following commutative diagram 

— '2d d 7r(jXid 



M /2d x M ,2 d Mi = Mi 







X' i2d) x pic2d Pici 



ctxid 



Pic 2d x p . c2d( Pic£ = Pic^ 
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and that \N E ~ n' n] \N^ n . 



3. Restriction of the Whittaker sheaf 

Let m > 1 be an arbitrary integer and define the Whittaker sheaf \N E,m by the 
same formula (2.7) but replacing n by m (where E is still of rank n). Consider the 
following commutative diagram 




and let u m = (/x m ,7r m ), 5 m = S' m o v m . Proposition 2.8 will follow from the following 
two propositions and this section is dedicated to prove the first one. 

Proposition 3.1. For any local systems E on X' of rank n and L on X of rank 1, 
m > 1, d > 0, the natural map 

C\ (w| d m H D*A L ) — ► 8 m w* m (w| d m m d* A L ) 

zs an isomorphism. 



Proposition 3.2. Let notations be as above, there is a canonical isomorphism 
*m< fw|L H T>*A L ) (As(E) ® L) W) [2d] 



on X^, w/iere = (As(£) (8 L) ( } C (As(E) (8 L)f J C • • • C (As(£) <g> C ■ ■ ■ 
filtration of (As(£) ® L)W snc/i «wrf (As(£) ® = (As(£) ® L)( d ). 



a 



3.3. Stratifications- 1. To proceed, we recall stratifications defined on Q'^ and M Q^) 
respectively. 

For any A G A^ os , let Q ^ be the stack classifying the data (^#', (s^), (D-)) where 
^T' is a vector bundle on X' of rank m, (D[, D[ +D' 2 , D[ + ■ ■ ■ + D' m ) G X' x cf. 0.8, 
and s[ (i = 1, m) is an inclusion of vector bundles 

s[ : fi^-^+'^^^^i + • • • + #0 — > A 4 ^' 

such that (s' l5 s^) satisfies the Pliicker relations. We have a natural morphism 
Q ^ — >• Q ^. It was shown in [2] that each q'^ becomes a locally closed substack 
and they together form a stratification of Q ^ for all A G A^ . We have a natural 
morphism s' : Q ^ — >■ X /A by remembering (Z)^, + D' 2 , Z^ + • • • + D^) and define 
Q m ? = ^ m X x /A wn i cn are closed substacks of Q m , s' ? = s'Iq'* for ? = eff , +, — . 



11 



YIFENG LIU 



For any A € A^ pos , let be the stack classifying the data (^#, (sj, (-Dj)) where 
j& is a vector bundle on X of rank m, (D\, ...,Di + • • • + D m ) € X x , and is an 
inclusion of vector bundles 

Si : g, H 1- Dj) — > 

such that (si, s m ) satisfies the Pliicker relations. As in the previous case, we have a 
natural morphism — > such that becomes a locally closed substack, and 



A 



they together form a stratification of ^Q^- We have a natural morphism s : — > X 
and define ^Q^? = /i^i x x A -^n which are closed substacks of ^Q m , S? = s| nA for 
? = eff,+,-. 

The pullback under /i again induces a natural morphism /Z A 



A 

'm 1 M^m,? 



■ „Q A , 



Q'i\ and 



hence : ? — > Q ^ ? x m ,mM^ for ? = 0, eff, +, — which are all closed embeddings, 
such that the following diagram commutes 




X 



- (/j A ,aosym A ) 



sym' 



s'xo 



X'? A x p . 2d Pic d 



sym 2A xid 
x ,{2d) 



Xp ic 2d Pic 




0' X0' 



For A G A^_, the stack Q'^_ 



will be nonempty and it equivalently classifies the 



data {{JKD, (r-), (£>•)) where = C .< C • • • C 



is a complete flag of 



sub- vector bundles of D[ is as above but with < D[ < ■ ■ ■ < D' m , and r[ is an 
isomorphism 



i-1 



for i = 1, ...,m. Let ev : Q ^ _ — >■ G aj fc be the morphism sending the above data to 

the sum of m — 1 classes in Ext 1 (il^7*~ 1 (^)i ^x^C^i)) — G a ,fc corresponding to the 
pullbacks of the extensions 







/-l 







'i+l 



under the inclusion 0™7 t_1 (£>-) ^7 l_i (£>- +1 ) for i = l,...,m - 1. Finally, let 



AS^ = ev A *AS,/, be a local system on Q 



3.4. Pullback of Laumon's sheaf. For A 6 A^_, the scheme X^_ classifies (D\,...,D 
where Dj is a divisor on X of degree Aj and < D 1 < ■ ■ ■ < D m . Let c A : X* ->■ Coh$ 



<m 
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be the morphism sending (JD\, D m ) to 

^(DO/n™- 1 © Qx~ 2 ® =^p2)/o^- 2 ® jsf„ © ■ ■ ■ © %™- x (p m )i%™- x . 

Similarly, we have a morphism c /A : X /A . — > CoIi'q <m sending (D[, ...,D' m ) to 

n^rHD'^/n^r 1 © o™r 2 (£> 2 )A^r 2 e • • • © ^r 1 ^)/^ 1 - 

By [12] Theoreme 3.3.8, the restriction c /A *Lau^ of Laumon's sheaf has maximal (possi- 
ble) cohomological degree (with respect to the usual t-structure) 2d(X), where d(X) := 
Y^iLi( m ~ ^ s highest cohomology sheaf 

E x :=H M ( A V A *Lau! 

does not vanish if and only if Ai = • • • = A m _ n = (which is an empty condition if 
m <n). See [12] for a description of the stalks of E^_. The following proposition proved 
by Lysenko [15] Proposition 2 will play a key role later, of which the proof uses the 
geometric Casselman-Shalike formula for general linear groups cf. [8], [16] and [17]. 

Proposition 3.5. 2 The restriction Wg m |n'A supported on Q ™ «is further 

restriction to the later one is isomorphic to 

AS^ ® s'_*£J A [2g m +md- 2d(A)]. 



The following simple lemma will be important in the later proof. 
Lemma 3.6. Let & be a coherent sheaf on X and consider the paring 

ev^ rExt^.n*) x H (X',//^) Ext 1 ^*^, Q X ') * H°(X',//J*") 
— ► H 1 (A A/ , Ox') — G a ,ifc- 

T/ien 

pr 2! ev^*AS v , = i° 2 dimH°(X, &)] 

where pr^ is the projection to the i-th factor, i\ : Ext 1 ( I ^", Six) ^ Ext 1 (//*^", fix') anc ^ 
£_ : Hom(«£f M , <— >• H°(X', //*J^~) are closed embeddings induced by fj,. 

Proof We only need to show that the orthogonal complement of i^Ext 1 (^", Six) C 
Ext 1 (//*^, Ox') inside H°(A A/ , /jl*^) under the canonical pairing is i°_Hom(^f^,^ r ). 
The paring between these two subspaces is zero due to the fact H°(X,JSf M ) = 0, and 
they are orthogonal complement of each other because they have the complimentary 
dimensions since B°(X', fj*&) = H°(Jf, © Hom(i^, ^). 

□ 



A similar result on the restriction of Whittaker sheaves on different strata is proved in [9] Propo- 
sition 4.12. 
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3.7. Proof of Proposition 3.1. By Proposition 3.5, it amounts to prove the following 
assertion. For p E A^_, the direct image (s' x 5')iAS^ vanishes unless p = 2A and then 
the natural map 

(*' x OO.AS^ — ► (*' x t>')-^,^*AS^ 

is an isomorphism. 

For j = 1, ...,m, we introduce the stack ■ classifying the data 

where C p* j&\ C • • • C p*^j C C • • • C ^' m _ x C is a complete 

flag of sub-vector bundles of p* where is a vector bundle on X of rank m; 

< p*D 1 < ... < /x*^ < £)J. +1 < • • • < Dm is in X'^ (p*D := p~ l {D)) and 

r< : n^"* OJS^-^A) ^ JtilJti-i for » = l,..,j, r< : fi^T*^) ^> 1 for 

1 = j + 1, m (^#o = 0, ^4 = p*^). Hence y p m j is empty if 2 f pj for some 1 < i < j. 
We have the following successive closed embeddings 

•VP ^> <pP <^-v ■ ■ ■ <-+ 'pP — O lp v nJ 

J m,m ~ J m,m— 1 ^ ~ ^m,l — ^ m,- M'^J Jvl -m 

and Jm,m is empty unless p = 2 A in which case = M^m- Write v p m ■ for the 

inclusion CP^ • Q ^ x M ,2d M^. We prove successively that the natural map 

is an isomorphism for j = 1, m — 1. 

In fact, let %^ be the stack classifying {Jf , MO£7+i5 ( D i)i=v ( D i)™j+V 
where C ^K'+i C • • • C ^V^-i c is a complete flag of sub-vector bundles of 

p*u¥ where j¥ is a vector bundle on X of rank m — j, (-Dj)^ =1 , (A;)ELj+i are as above 
and rj : WgT* A'+i/A' W = 0, = pfjV). The closed substack tt^ ^ 
is defined by the condition that : f2™r J_ (-D^- +1 ) coincides with the 

pullback of f J+1 : 0, r £~ 3 ~ <g) Jf^(Dj + i) ^> for some divisor -Dj+i on X with 

p*Dj + i = D'j +1 . There are natural morphisms f : J 1 ^- — )• ft^j and f : "^mj+i ~* ^Cj 
via taking the quotient by which are generalized affine fibration (c/. [15] 0.1.1 for 
the convention). We have the following commutative diagram 

17 mj'+l 




in which the square is Cartesian. Applying Lemma 3.6 to the vector bundle JV <g> 
^-m+j+i (g ) ^"J+ 1 j one eas ily S ee that fii/^ -*ASfl is supported on R^j, and our asser- 
tion follows. Hence Proposition 3.1 is proved. Moreover, it has the following corollary. 
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Corollary 3.8. Let E and L be as in Proposition 3.1, the complex d m \ v * n (W^ m Kl 

f *Ai)[— 2cZ] on XW is placed in degree zero. It has a canonical filtration by constructible 
subsheaves such that the direct sum of all graded terms 

grc) m , ! <(W| d fn Kc)*A L )[-2d]^ (symV*^)®^ 

In particular, gr f) m! z4 (W| d m B d*A L )[-2d] ~ gr 5 n! f*(W| d n B Q*A L )[-2d] for m>n. 

Proof. Since the morphism s : _ — >■ is a generalized affine fibration of rank 
q m + md — d—2d(X), the assertion follows from Proposition 3.5, 3.7, Lemma 3.6, and the 
Cousin spectral sequence for the computation of the !-direct image via stratifications. 

□ 

4. Reduction to the moduli of torsion flags 

4.1. More stacks. Recall that we have stacks Q „ <— » Q „ and : Z m = Q ^ x Bun < ro 
Mod'^ -> Q'^. Define = 0'°, X Bun ' m Mod'^ <-> Similarly, we define tl Z d m = 
A. x B un m Mod^. 

Define S„ to be the stack classifying the data {jY , (^/), (V0) where is in Coh^, 
= I^CI(C-C<C and : fi^, - * ^> M'JJC^ for i = l,...,m 

are isomorphisms. Define to be the open substack of C^f by the condition that 
c/K is locally free. Then C°^f is naturally identified with Z'^ x Bun / m Bun m . Now 
consider the subfunctor of G^!~ defined by the following way. For any scheme S, 
(J*S, {^Ls)i ( r i,s)) G Hom(5, e^) is in Hom(5, ^ ) if there exists (ytf^s, (n,s)), where 
^i t S is a vector bundle on S x X of rank i, = ^#0,5 C -#1,5 C • • • C ^m t s C and 
r i,5 : ^ < ^ ) -^> _1 ) -^i,S / ,S for i = 1, ...,m are isomorphisms, such 

that r 4 ' 5 : G s M 0™r* ^> ^is/^Li.s coincides with fj,*r i>s on S x X' - (id x n)~ l T 
for a closed subscheme T of 5 x X which is finite over S. We have 

Lemma 4.2. The subfunctor yj™ is a closed embedding, hence is an Artin 

stack. 

Proof. An object (^5, (^( s ), (r- s )) £ Hom(5, C^f) will induce maps 

4 S : G s B O^" 1)+ ""' +(m_0 — > AV*^ = ^* A' </f 
for i = 1, m. Let 

Then (^5, (^ 5 ), (r^)) e Hom(5,y^) if and only if the support of s^-(-l)^s^ 5 <T 
is a closed subscheme of 5 1 x X' finite over S for i = 1, m. Since is flat over S", it 
is locally free outside a closed subscheme of 5 x X finite over S. Hence the assertion 
follows from [15] Sublemma 4. 

□ 
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For (,jV s , (^(s)> ( r 'is)) G Hom(5,y^), the induced map 



si :^ S B!] 



m(m— 1) 
2 

X' 



det . 



^ det h*JV 



satisfies that s\ 



(-1) 



m(m — 1) 



si) 11 . Hence the zero divisor of s* — S locates in the 



closed subscheme S x of S x X'^ 2d \ It induces a morphism d m : y 2 ^ — > X^. 
Define T 2 J = C o2d n y 



2c/ 



y 2 ^ which is an open substack naturally identified with 

to y o2 ^ 

id x m . 

For j = 0, m, we define a subfunctor y^ Ui 
data ((^ / )f=i,W)i 

=1 ) is a pullback of ((^#j)] =1 , (r- 
above sense. Then we have a successive closed embeddings 



2>'m X Q'2d /iQm- Moreover, the restriction of d 



x Q'2dM^m coincides with 



^ by the condition that the partial 
i=i) ( on the whole S x X) in the 



y 



2(7 



' <Jm,rn— 1 



y2d 
9m,0 



y 



2</ 



2d 



and natural morphisms y 
eralized affine fibration. Let y 02 ^,,- 
y°m m i s naturally identified with 



^m-j via taking the quotient by 



_ po2d p. u2d 

77 f 



, which are gen- 
^mj ^ e the open substack, then 



4.3. Iterated modifications. We have a natural morphism y 2 " C 2 ^ — > Coh' 2 ^ sending 



(^, «), (r0) to \?JriJ(' m , and define y 2 ^ = ^ x Coh ,, d Fl'§ d , y°^ = y 



m 
o2d 



Fl' 2d . Denote c m : y 2 ? ^ Coh '2d 



m A x /(2d) 



and c m : y 



2(/ 



F1 /2d 



X X'( M > 



-Coh'g d 



the induced morphisms from d m . We have the natural projection p m : y 2 ^ — >■ y 2 ^ 
Moreover, for j = 0, m, we define y 2 ^- = V^- x^m y„ and p m j : y 2 ^- — 
natural projection. We have the following commutative diagram 



y^- the 




u2d r 11 

"ro,3 




Coh /2d x x , (2d) 



div^ ffl xid 



pr 




Similarly, we define y o2 fj = y 



2ri 



Xu2d 



y°m which is a closed substack of y°™ and an 



open substack of y 2 ^ ■. We have restriction of morphisms u° 



u° n° n° d° 



d^ and d^. But now the restriction of c m (resp. c m ) to y o2 ^ (resp. y°^f) will 
factor through Fl' 2 , 6 ^ x Coh /2d CoIiq (resp. Coho <m ), and we define c° m (resp. c£j to be 
the restriction but with this new target. Let c° m ■ = c° 



o u° .. We will have a similar 

''hj 
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diagram as above which we omit. In particular, the stack V ^ m * s naturally identified 

withX X Oahg ( Coh 0><Coh- PI'* 1 ). 

As for Q'^, we define in the same way the evaluation map ev 2d : y 2 ^ — > G^., 
and AS 2d = ev M *AS</,, AS 2d = p^AS 2d which are rank-one local systems cf. 2.6. The 
*-restrictions of AS 2a! to y 2 ^- and y o2 ^- are still denoted by AS 2d . Then we have 

Proposition 4.4. TTie morphisms is o/ relative dimension < q m + (m — l)d, and i/ie 
natural map of the highest cohomological sheaves 

H 2(g m + (m-l)d) d^ASj* — ► H 2 ^™^™" 1 ^ d^ jm! Q7 

zs an isomorphism. 

Proof. The estimation of the relative dimension will be accomplished in Lemma 4.7. 
For the second assertion, we inductively prove that the natural map 

TT2(<jm+(m— jo ac2o! . Tj2(<j m +(m-l)cZ) jo Ac2d 

is an isomorphism for j = 0, ...,m — 1. Consider the commutative diagram 

Uo2ci C Uo2d C >. u2d 

O mj+1 O ni,j "mj 




u2d C y2d 

"m—j,l "m—j 

where the square is Cartesian. The morphism k is a generalized affine fibration of rank 
Qm — Qm-j + jd and hence the natural map 

jj2(q m -9 m _j+j'd) K °AS2d ^ jj2(g m -g m _j+jd) K! /\S 2d 

is an isomorphism over the image of K°. Applying Lemma 3.6 to the coherent sheaf 
J^/JKj ® ft" m+i+1 ® J2£~ i+1 , we see that H 2 ^ m-9 "» _ -' +J '^ K,AS 2d is supported on the 

closed substack x- Hence, the natural map 

is an isomorphism. To conclude the desired isomorphism at the beginning of the proof, 
we only need to apply Lemma 4.8 for d m -j. Now the second assertion immediately 
follows since ASB, d |y o2d = Qi by Lemma 3.6. 

□ 

Corollary 4.5. Let E and L be as in Proposition 3.1, then we have a canonical isomor- 
phism 



fm< (wi d m ^*A L ) [-2d] (H- 2d div d < m! ^Lauf) 

o/ constructive sheaves on X^ d > . 
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Proof. By the above proposition, we have canonical isomorphisms 



Jj2(q m +md— d) ^ 



c* fSpifKIL^ 



div x2d *E m 2d 



L (d) 



L {d) 



TT2(q m +md-d) jo ~ * 
±x J Ji ! u m,m! L m,m 



x2d* 2c2 



jo2d 

' 771,771 



But the morphism d^ m : V 



o2d 

m,m ■ « m,m 



X' 



2d 



- x ,(2d) 



is the composition of 



\so2d 



™i2d 
rl 0,<m 
,2d. 



T?]'2d 

rL 0,<m x Coh'g d 



x Coh /2d Cohg and the natural morphism div< d m 

Cohg -> X' zc x v/(2<i) Af W , in which the first one is smooth and surjective with connected 
fibre of dimension q m + md and the second one is of relative dimension < —d by Lemma 
5.2. Hence 



rr2(q m +md-d) jo ~ * ( (a ■ X 2d* p 
M P r ! a m,m! C m,m U aiV ^ 



I 2(2 



H- M div^ m! ^Spr| d )®LW 



where all these sheaves carry natural actions of &2d and the isomorphisms are equi- 
variant under these actions. Taking &2d invariants and by Corollary 4.5, we get the 
desired isomorphism. 

□ 



Z'fn x Coh / d FI'q introduced 



x g'd Q m is stratified by locally 



4.6. Stratifications-II. We recall a stratification on Z"j: r 
in [15] Section 4.3. For A G A^ eff , the stack 2/£ 
closed substack Z/^ for e G (in the notation of [15]), where e 
matrix with entries being non- negative integers such that e| = 1 for j = 1, d and 
= Aj for i = 1, ...,m. Put X' e = n^--^ 76 *, then there is a natural morphism 



e~j ) is an d x m 



following. 



Xj^,a Q m is an affine fibration of rank (i(A). Now it is easy to deduce the 



Lemma 4.7. The morphisms is of relative dimension < q m + (m — l)d. 
Proof. The restriction of d^ to the stratum Z/^ 



x q'2a M^m fact 01,8 as 



cyle v f)-^ 
~ m A q'2a /u-'m 



if 

generalized affine fibration of rank q. 



x 



X' 



where the first morphism is an affine fibration of rank d(2X) and ^Q^ — > X x is a 

l)d — 2d(X). Hence the lemma follows. 



(m 



□ 



Now we estimate the relative dimension of d m . There is a stratification of by the 
degree of the maximal torsion subsheaf & of jV. We define a locally closed subscheme 
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^m' 2d ' of y% for < d! < d classifying the data (jV , (J?!), (r0) G plus a flag of 
subsheaves M' m = jYq C jV{ C • • • C J/ll A = and an exact sequence 

0^, 

of ^x-modules such that is a torsion sheaf of degree d' and "V is a vector bundle 
of rank m. Let = .jVJ fl /x*^ for 2 = 0, ...,2d, then the successive inclusions of 

torsion sheaves = ^ C C - - - C = define a P°i nt of Fi 'o d ' x Coh'g d ' Con o 
and an element c € C^d > which is the set of subsets of {1, ...,2d} of cardinality 2d'. 
Hence c determines a locally closed subscheme of ^m' 2d and for each c, a morphism 
— > FI'q^' x Coh ,2d' Cohg on one hand. On the other hand, the quotient sheaves 

yf = Jf'llii*^ and the induced flag define a point of y o2 £~ 2d ' . They together define a 
morphism — > ^Fl'p*' *- Coh i2d' CoIiq'^ x^ o2 ^~ 2d ' which is a generalized affine fibration 
of rank md'. By Lemma 4.7, T^~ 2d ' X' 2d ~ 2d ' x x , (2d _ 2d , ) X^- d "> is of relative 
dimension < q m +(m—l)(d—d'), and by Lemma 5.2, Fl'^' x Coh , 2d ' Cohg — > X' 2d x. x ,$d?) 
X( d '} is of relative dimension < —df, which together imply the the following. 
Lemma 4.8. The morphisms d m is of relative dimension < q m + (m — \)d. 



5. Direct image of Laumon's sheaf 

5.1. Stratifications- III. A point of Fl'g d x Coh /2d Cohg is given by a torsion sheaf & on 
X of degree d and a complete flag of torsion sheaves = J^q C C • • • C = 
= a *fj,*^ of n*&. Let J^- = J^/ n C7*«^j, then = cr*^- as subsheaves of 
\x*& . As in [15] Section 6.3, define from the co-Cartesian square 



for z,j = l,...,2d, and %j = Let iy = degSfjj. Then it is easy to see 

that i = (tij) is an element in 1.2d cf. 1.5. In this way, FI'q^ x Coh /2d Cohg is stratified 
by locally closed schemes F1' for t € T2d- For each i, the natural morphism Fl q — > 
11^ Coh^' by sending (J? (J?/)) to (^) i<i is a generalized fibration of rank 0. Since 
Coh'J — > X' — > X is of relative dimension —1, we have the following lemma whose 
second assertion follows from the same argument of [15] Lemma 11. 

Lemma 5.2. The morphism 

div 2d := (div xM o q') x div d : Fl' 2 d x Coh ,2 d Coh^ — > X ,2d * x , m X® 

is of relative dimension < —d and we have a canonical isomorphism 
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where n is defined in 1.5. 

5.3. Proof of Proposition 3.2. By the above lemma, we have an isomorphism 

(H- 2d div1^Sprf) ® pr, ( (e 81 M H ® n,Q^) 

which is S2d-equivariant. Taking invariants on both side and by Lemma 1.6 (3), we 
have 

(H-^divl^Lauf) <g> lM> ^> (As(£) ® L) (d) . 
By Corollary 4.5, we have 

f m! ^(w| d m Kc)*A L ) [-2d] 




^H- M div| m! /i^Laufj (H- 2d div1/^Lau| d J 

(As(£)®L) (,i) . 



Denote (As(£) ® L) C (As(£) ® L) (ci) the image of 5 m! z4 (w| d m H c)*A L ) [-2d]. 

Then we have a filtration = (As(J3) ®L) { Q d) C (As(£) C • • • C (As{E) ®L)$ C 

• • • of (As(£) ®L)M such that (As(£) ® = U ro (As(.E) By Corollary 3.8, 

the filtration becomes stable when m > n. Hence the proposition is proved. 

Remark. In fact, it is not difficult to see that the inclusion 

3m!«4 (W| d m H t)*A L ) [-2d] ~ (Ab(E) ® L)W 
^(As(£) L)$ ~ 9 mq ^ (W| d m , B J)*A L ) [-2d] 
for m < m! is compatible with the canonical filtration in Corollary 3.8. 

Acknowledgements. The author would like to thank Weizhe Zheng for helpful dis- 
cussions. 
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